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Background / Context: 

Description of prior research and its intellectual context. 


A plethora of laboratory studies have shown that including the study of worked examples 
during problem-solving practice improves learning (Sweller, 1999; Sweller & Cooper, 1985). In 
addition, explaining instructional material has been shown to improve learning by forcing 
students to make their new knowledge explicit (Chi, 2000; Roy & Chi, 2005); asking students to 
explain examples thus further improves student learning over having them simply study 
examples (e.g. Renkl, Stark, Gruber, and Mandl, 1998). 

While most worked-example research focuses on the use of correct examples, recent 
work suggests that asking children to explain a combination of correct and incorrect examples 
can be even more effective (e.g., Siegler & Chen, 2008). The benefit of explaining errors is 
twofold. First, it can help students to recognize and accept when they have chosen incorrect 
procedures, leading to improved procedural knowledge over practice alone or correct examples 
plus practice (Siegler, 2002). Second, and perhaps more important, it can draw students’ 
attention to the particular features in a problem that make the procedure inappropriate. For 
instance, consider an example in which the equation 3x - 4 = 5 is incorrectly solved by 
subtracting 4 from both sides and resulting in a next problem state of 3x = 1 . By explaining how 
the procedure led to an incorrect answer, students are forced to both accept that the procedure is 
wrong, and to notice that the negative sign that precedes the 4 makes it inappropriate to apply the 
strategy. This can help the student replace faulty conceptual knowledge they have about the 
meaning of the problem features with correct conceptual knowledge about those features; the 
acquisition of accurate, deep features with which to represent problem situations is key to 
building expertise (Chi, Feltovich, & Glaser, 1981). Consistent with this assertion, the 
combination of correct and incorrect examples has been shown to lead to improvements in both 
conceptual understanding and procedural skill in Algebra compared with procedural practice 
alone (Booth et ah, in revision). 

The combination of correct and incorrect examples is thought to be beneficial because 
the incorrect examples help to weaken faulty knowledge and force students to attend to critical 
problem features (which helps them not only to detect and correct errors, but also to consider 
correct concepts), while the correct examples provide support for constructing correct concepts 
and procedures, beyond that embedded in traditional instruction. It seems clear that both types of 
support are necessary, but what if extra support for knowledge construction is achieved through 
other types of innovative classroom practice? In that case, would it still be optimal to provide a 
combination of correct and incorrect examples, or would providing incorrect examples alone 
suffice for improving student learning? In the present study, we test the contribution of correct 
vs. incorrect examples in the context of such support for knowledge construction— guided 
problem-solving practice with the Cognitive Tutor, a self-paced intelligent tutor system which 
provides students with feedback and hints as they practice (Koedinger, Anderson, Hadley, & 
Mark, 1997). 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 
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The present study examined two primary research questions. First, which type of examples is 
best for improving student learning in Algebra? Second, does the optimal type or combination of 
examples differ for improving different facets of knowledge of linear equations? 

Setting: 

Description of the research location. 


Data collection took place in five classrooms recruited from a single middle school in the mid- 
western United States; Twenty-nine percent of students who attend the school receive free or 
reduced lunch. All data collection took place as part of regular classroom activities. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features , or characteristics. 


Sixty- four eighth-grade students (29 females, 35 males) in Algebra I classrooms using the 
Cognitive Tutor software participated in this study. The ethnicity breakdown was as follows: 
37% Caucasian, 52% Black, 1 1% multi-racial. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

In each condition, 12 computerized examples were interspersed amid problem-solving 
practice in the Two-Step Linear Equations unit of the Algebra 1 Cognitive Tutor. Each example 
depicted either a correct or an incorrect solution step for solving a two-step equation; whether the 
example was correct or incorrect was clearly indicated. 

(INSERT FIGURE 1 HERE) 

The interface first asked students to explain what happened on each side of the equation 
by building a phrase from a series of pull-down menus (i.e., “Added”;“-3”). Students received 
feedback from the tutor on whether their answers were appropriate, and had to keep trying until 
they answered correctly. Then, the student was asked to build a sentence to describe why the 
step was right or wrong, depending on the nature of the example. Students again received 
feedback from the tutor and had to continue trying until they provided an appropriate 
explanation. 

In the Correct Only condition, students explained 12 correct examples, and those in the 
Incorrect Only condition explained 12 incorrect examples. Students in the Both group explained 
6 correct and 6 incorrect examples. Because students were allowed to complete the unit to 
mastery, and thus take as long as they needed, the total study time varied for each student; 
however, most of the students had completed the study within 4 weeks (8 class sessions). 

Research Design: 

Description of the research design. 

Individual students within the classrooms were randomly assigned to the three conditions 
(Correct Only (N = 22), Incorrect Only (N = 21), and Both (N = 21)). Students completed a 
pretest and posttest surrounding completion of their Tutor unit. The pretest and posttest were 
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identical, and each included four measures: Feature Knowledge, Encoding, Equation-Solving, 
and Flexibility. 

Feature knowledge questions measured students’ understanding of concepts that have 
been identified in previous research as crucial for success in Algebra; it was comprised of 37 
items that measured student knowledge of three critical features: the meaning of the equals sign 
(e.g., what does = mean?), negative signs (e.g., is 4x - 3 equivalent to 3 - 4x), and like tenns (is 
6 a like term for 6c?). 

Encoding of problem features was measured using a reconstruction task (e.g., McNeil & 
Alibali, 2004) in which students were presented with an equation for six seconds and then asked 
to reconstruct the problem from memory immediately after it disappeared from view. Students 
completed this task for a series of six equations with different structural formats and different 
placements of key problem features (e.g., 4x = 9 + 7x - 6; p - 5 = -2p + 3). 

The equation-solving measure tested students’ ability to effectively carry out procedures 
to solve six equations: three isomorphic problems (i.e., problems structured like those they were 
trained on) and three transfer problems (i.e., problems that include additional features or alternate 
structures from those they have been trained to solve). 

Finally we evaluated students’ flexibility in problem solving by measuring their ability to 
recognize and choose an effective strategy when solving an equation. Students were asked to 
solve three equations using two different strategies. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

Paper and Pencil tests were scored for correctness. Each student received one score for 
percent correct on each of the four measures, as well as two composite scores— Conceptual 
Knowledge (feature encoding and feature knowledge) and Procedural Knowledge (equation- 
solving and flexibility). 

Findings / Results: 

Description of the main findings with specific details. 


To detennine whether there was any effect of condition on the conceptual understanding 
measures, we conducted a 3-level (Condition) MANCOVA on posttest feature knowledge scores 
and posttest encoding errors on conceptual features, and posttest encoding errors on non- 
conceptual features, with overall pretest conceptual understanding scores included as the 
covariate (see Figure 4, left 2 columns). The multivariate effect of condition was significant, F( 6, 
120) = 3.1 1, p < 0.01, r]p = 0.13. Significant univariate main effects of condition were found for 
conceptual features (F(2,64) = 5.30, p < 0.01, q p 2 = 0. 15) and conceptual encoding errors 
(F(2,64) = 3.56, p < 0.05, q p 2 = 0.10). Follow up pairwise comparisons with Bonferroni 
correction indicated that for the conceptual features measure, students in the Combined condition 
(M = 72%, SD =15%) scored significantly higher than students who received the Correct only 
condition (M = 53%, SD = 26%, p < 0.01). In addition, students in the Incorrect only condition 
made fewer conceptual encoding errors (M = 2.0, SD = 2. 1) than students in the Correct only 
condition (M = 4.4, SD = 4.2; p < .05); the comparison between the Combined and Incorrect 
only conditions was not significant. The univariate main effect of condition on non-conceptual 
encoding errors was marginally significant F( 2, 64) = 2.55, p < 0.10, q p 2 = 0.08), however, 
follow-up pairwise tests revealed no significant differences between individual conditions. 
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A parallel 3-level (Condition) MANCOVA was conducted on posttest isomorphic 
equation-solving, transfer equation-solving, and flexibility scores, with overall pretest procedural 
fluency scores entered as a covariate (see Figure 4, right 3 columns). The multivariate effect of 
condition did not reach significance for the procedural fluency measures, F(6, 120) = 1.38, p = 


.23, r| p 2 = 0.06. 


Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


Students performed best after explaining incorrect examples; in particular, students in the 
Combined condition gained more knowledge than those in the Correct only condition about the 
conceptual features in the equation, while students who studied only incorrect examples 
displayed improved encoding of conceptual features in the equations compared with those who 
only received correct examples. 

No differences were found between any of the conditions on any of the procedural 
measures. This indicates that the specific types of examples provided may not have much 
influence on procedural learning when they are combined with guided practice. 

The present study was the first to test whether a combination of correct and incorrect 
examples is more beneficial than incorrect examples alone, and suggests that receiving incorrect 
examples can be beneficial regardless of whether it is paired with correct examples. This finding 
is especially important to note because when examples are used in classrooms and in textbooks, 
they are most frequently correctly solved examples. In fact, in our experience, teachers generally 
seem uncomfortable with the idea of presenting incorrect examples, as they are concerned their 
students would be confused by them and/or would adopt the demonstrated incorrect strategies for 
solving problems. Our results strongly suggest that this is not the case, and that students should 
work with incorrect examples as part of their classroom activities. 

Though inadequate conceptual knowledge of features in algebraic equations hinders 
students’ success in Algebra (Booth & Koedinger, 2008), the present study demonstrates a 
successful plan for intervention: Provide examples along with guided practice problems to 
improve conceptual understanding. Specifically, exposure to incorrect examples which target 
typical student misconceptions about problem features and problem-solving approaches may be 
crucial for developing students’ conceptual knowledge. Viewing and explaining such incorrect 
examples may help students to both confront their own misconceptions as well as refine their 
understanding of the features in the problem; this may be especially so when students are guided 
to notice critical features in the problems, either by forcing them to explain what changed 
between steps or by prompting them to answer specific questions about the mistakes in the 
problem. 

Future research should examine whether the optimal combination of examples differs for 
students with varied levels of background knowledge; such results would not be unprecedented 
(GroBe & Renkl, 2007; Kalyuga, Chandler, & Sweller, 2001). Systematic exploration of such 
individual differences in future studies is necessary to identify exactly when each type of 
examples are most beneficial for individual students. This knowledge would have the potential 
to improve instruction on equation-solving for students in the Cognitive Tutor curriculum, as 
well as to inform all teachers of example-based strategies that can be used to differentiate algebra 
instruction for their students. 
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Figure 1 


1 - Two-Step Linear Equations 

2 - Correct Example AC 1 

Table of Contents^ Lesson j fProblems 


Jermain was asked to solve the following equation for x, and he made a GOOD 
first step to solve the problem. Look at his first step. 

3 = 2x - 7 
10 = 2x 

What did Jermain do? 


On the left side he... 


On the right side he... 


Choose one... ▼ Choose one... ▼ 

Choose one... ▼ 

Choose one... ▼ 



Choose one... 




Added 



Add 

Subtracted 

Add 



Multiplied 

Divided 



Why is that a GOOD step for Jermain to take? 

Did nothing 



Please answer both why it was a VALID step and a HELPFUL step. 



1 


It is valid because ▼ Choose one... ▼ 

Add 


It is helpful because ▼ Choose one... ▼ 

Add 
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